Video streaming over reliable channels has become commonplace. Challenging technical problems, however, remain to be solved in order to consistently support video streaming over unreliable channels such as channels having high loss rates. It can be difficult to provide a reasonable quality of service at high loss rates because, typically, video streaming applications are designed to operate in low loss environments. Therefore, many video streaming applications are not exceptionally error-resilient nor network adaptive in high loss environments like wireless networks.
In low loss environments, it is relatively simple to correct or retrieve information lost in the process of video streaming. The easiest and most common method to retrieve lost data is to simply send a message from the receiver to the transmitter requesting the lost information, and then to have the transmitter transmit the requested information to the receiver. Of course, sending the message from the receiver to the transmitter and transmitting the requested information from the transmitter to the receiver takes time.
The time required to execute the easy method of information exchange between the receiver and transmitter described above is what makes the easy method unattractive for use in high loss environments. High loss environments do not have the extra time to carry out the easy method. Wireless and other high loss environments are often utilized for “live” event broadcasts, and such broadcasts may be transmitted over a considerable distance. Accordingly, any additional delay between the transmitter and the receiver in a high loss environment can add or add to choppiness to the video broadcast. Thus, sending a message from the transmitter to the receiver requesting lost information and having the requested information transmitted to the receiver generally is not an option for retrieving lost information in a high loss environment.
To improve video broadcast where video streaming occurs in high loss environments, efforts have been made to minimize information loss by error control on the transmission side of the environment. For example, attention may be given to using appropriate video coding algorithms as part of error control. Unfortunately, most of the current video coding algorithms were developed for relatively error-free environments. High loss environments are often considered unreliable because of high error rates and highly variable load capacities. Thus, the current video coding algorithms are generally not particularly useful in high loss environments.
A constraint on error control at the transmission side of high loss environments has been addressed briefly above in the discussion of the problems associated with the use of the easy method to retrieve lost information in high loss environments. Delay in transmission of information between the transmitter and receiver in a high loss environment can add or add to choppiness of the video broadcast. For example, many wireless video streaming applications are highly sensitive to end-to-end delay. Thus, stringent delay requirements constrain any error control strategy for high loss environments.
An attempted solution to the problems of high loss environments and video streaming's sensitivity to end-to-end delay includes forward error correction (FEC). FEC allows a user to address a potential transmission problem prior to transmitting the data. FEC provides coding that serves as extra protection for the transmitted information. The provision of extra coding, however, can be a complex and time consuming endeavor. Moreover, the extra coding may cause the use of much more bandwidth than without the extra coding.
A popular method of FEC that addresses the issue of conserving bandwidth is termed unequal error protection (UEP). In sum, the more important bits of the bitstream are protected using a stronger FEC code than the less important bits.
More particularly, commonly used video compression algorithms produce bits that contribute quite differently to reconstructed picture quality. Not all of the transmitted bits in a compressed video sequence have equal importance. The theory of UEP is that the more important bits of the transmitted video are protected at a higher level than the less important information.
A limitation of UEP, however, is that it may introduce delay that does not meet the restraints imposed by the stringent end-to-end delay requirements in high loss environments. UEP may introduce delay by its employment of complex algorithms to determine what level of protection is necessary to protect the varying bits of the bitstream. The use of the complex algorithms introduces delay.
More particularly, in the previously proposed UEP, the bitstream is divided into a large number of data fragments according to the importance of the bits. A different FEC code may be assigned to each data fragment. Finding the optimal FEC code for each of the data fragments may take a relatively long time. Thus, delay is introduced. Further, a large number of different FEC codes may be generated and a large amount of overhead information may be sent to the decoder for FED decoding. Although UEP algorithm provide finer quality levels, the UEP algorithms require long execution times to find the optimal FEC codes. This large amount of information and long execution times introduce delay. Given the stringent requirements in high loss environments, the introduced delay may be too much and render use of UEP inappropriate in high loss environments.
A category of methods utilized to reduce the effect of errors in high loss environments is referred to as multi-substream partitioning, which may include multi-substream frequency domain partitioning. Multi-substream frequency domain partitioning has been utilized for wavelet-based coders. In multi-substream frequency domain partitioning, the coefficients of wavelet transforms in a group of pictures (GOP) are divided into several groups, and each group is independently encoded. Each substream is protected using a UEP method and transmitted to a receiver. Thus, errors in one substream do not affect the other substreams.
Once the substreams are decoded, the receiver reconstructs the image from the several descriptions provided by the substreams. Therefore, a missing bit in one substream is not catastrophic because information from one or more of the other substreams may be used to estimate the lost bit.
An advantage of multi-stream frequency domain partitioning is that it has high source coding efficiency. A disadvantage, however, is that correlation between the substreams in multi-stream frequency domain partitioning is not high. Yet, a high correlation between substreams is important to the recovery of lost information at the receiver. Thus, the lack of high correlation in multi-stream frequency domain partitioning argues against its use in high loss environments despite its high efficiency in source coding.
Another method to reduce the effect of errors in a high loss environment is referred to as multi-path transmission. Multi-path transmission allows multiple substreams to be transmitted, respectively, across independent paths. Accordingly, delay or high loss rate in one path does not affect a substream transmitted over a different path. Multi-path transmission is utilized primarily within the moving pictures experts group (MPEG) community.
Advantageously, the methods of multi-substream partitioning and multi-path transmission may be combined. For example, substreams may be transmitted in respective independent paths to the receiver. One of the substreams encounters a high level of loss during the transmission. Another substream encounters the least loss of the transmitted substreams. The receiver may utilize the substream with the least loss to estimate lost bits in a substream that encountered the high level of loss.
Unfortunately, utilizing the methods of multi-substream partitioning and multi-path transmission, especially in the MPEG community, may add unacceptable levels of delay. The delay may be caused by use of relatively complex algorithms in the methods. Additionally, these methods may require transmitting a large amount of additional information to the receiver, which may add to bandwidth usage and also cause delay.
In the receiver side of a high loss environment, methods of error concealment have been used to reduce the effect of errors, and such methods have been used especially by the MPEG community. Error concealment utilizes the fact that most images are smooth to estimate lost bits within a substream. Additionally, error concealment uses one substream of two or more substreams to estimate lost bits in a different substream. A drawback to error concealment is that use of such methods of error reduction, especially with MPEG transmission, often involve the use of algorithms that may be complex, require a lot of information, and time consuming in operation. Thus, the methods of error concealment also may cause more delay than acceptable for video streaming in a high loss environment.
In sum, a need exists for methods and systems that address the drawbacks of video streaming in high loss environments. Methods and systems are needed that provide error-resilient and network adaptive video streaming. Additionally, the methods and systems must efficiently and rapidly protect a bitstream prior to transmission without requiring more bandwidth than necessary. Finally, there is a need for the development of methods and systems that provide efficient error concealment once a transmission through a high loss environment is received.